fix: split whiteSpace into multiple tokens to fix findCaretTokenIndex… #385
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fix #376
问题:当SQL存在换行时, 一个
token
会包括多行,findCaretTokenIndex
找不到tokenIndex
以下是问题SQL示例:
现象为换行处空格后没有补全:
![iShot_2025-01-06_19 15 42](https://private-user-images.githubusercontent.com/64318393/400405514-e7241ac1-2f84-477b-aa8e-5b7d2efbd524.gif?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MzkxMzM3NTMsIm5iZiI6MTczOTEzMzQ1MywicGF0aCI6Ii82NDMxODM5My80MDA0MDU1MTQtZTcyNDFhYzEtMmY4NC00NzdiLWFhOGUtNWI3ZDJlZmJkNTI0LmdpZj9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTAyMDklMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUwMjA5VDIwMzczM1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTcwMjI1NWMwZmQ0YjZjMjgzNGE3OGI1MmM5OTM1YjcwOTg0ZGYzNWNjMzkyZGU5OGNiNzE1ZjAwNDgwZjdjMjEmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.JfL1Axkj1GCDQXLpQ7oBcWl5FQKV1GvY4wS5w21b4G4)
问题原因:
![image](https://private-user-images.githubusercontent.com/64318393/400407213-20002398-b308-4b29-83b8-ffec859ce2a0.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MzkxMzM3NTMsIm5iZiI6MTczOTEzMzQ1MywicGF0aCI6Ii82NDMxODM5My80MDA0MDcyMTMtMjAwMDIzOTgtYjMwOC00YjI5LTgzYjgtZmZlYzg1OWNlMmEwLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTAyMDklMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUwMjA5VDIwMzczM1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTc0YzBmNzA3N2NiMjNkNzZlOTY5NjhiNjgyYzU1NDlhNTY4ZmI1NGE4OGVhYWE2MzFhNTJiZmMxYTk3MmM5MmMmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.gT69ffnRKlGeW2dSz4wBYgfynfs0FdDmYocyEkpK9Lc)
一个
token
内可能包含多个换行符,导致左边界判断有问题修复方式:
跟
Hive
保持一致,将每个空白符拆分到单独的token
中去